{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "35538e17",
   "metadata": {},
   "source": [
    "# 这一部分是词汇的笔记"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "190ae2c4",
   "metadata": {},
   "source": [
    "# 一.词汇表征"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ff6622b",
   "metadata": {},
   "source": [
    "<img src=\"./picture/21.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83cff2f2",
   "metadata": {},
   "source": [
    "在上一节中，我们可以使用$one-hot$编码来对数据进行操作，但是我们可以发现这个样子的话，每个\n",
    "单词之间并没有相关性。所以可以使用上述的方法来对已有的数据进行操作。\n",
    "\n",
    "我们可以规定一些gender，age,food,俩判断这些单词和上述单词的相关性，这个样子就可以得出每个词语之间的关系了\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff7ede7d",
   "metadata": {},
   "source": [
    "<img src=\"./picture/22.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39acba05",
   "metadata": {},
   "source": [
    "上面使用了一个算法叫做t-sne算法，可以将高纬度的数据映射到2维，这个样子就可以进行数据的可视化了。\n",
    "两个特征越接近，那么他们的相关性就越强，和聚类类似啊。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6029cee",
   "metadata": {},
   "source": [
    "# 二.使用词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc316c1b",
   "metadata": {},
   "source": [
    "词嵌入我感觉更像一种编码方式，而神经网络就是去训练这个编码，然后让他的相关性得到充分的训练即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ed50af",
   "metadata": {},
   "source": [
    "# 三. 词嵌入的特性"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb3b2499",
   "metadata": {},
   "source": [
    "<img src=\"./picture/23.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5230f0a",
   "metadata": {},
   "source": [
    "上面的图像，首先我们又man可以推理出woman，然后从king怎么推出queen的呢。\n",
    "\n",
    "首先将man的向量减去woman的向量，然后king减去queen的向量，发现他们很相似，由此可以推出，这是这个算法的基本思想"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c9e4ba1",
   "metadata": {},
   "source": [
    "<img src=\"./picture/24.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "655d4586",
   "metadata": {},
   "source": [
    "上面的这张图像讲述了怎么求解相似度的问题，首先构造一个上面的函数，然后让相似度函数最大化\n",
    "然后答案就是这个单词了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48de2faf",
   "metadata": {},
   "source": [
    "## 3.2 相似度函数（余弦相似度函数）"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "777e058c",
   "metadata": {},
   "source": [
    "<img src=\"./picture/25.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6334713",
   "metadata": {},
   "source": [
    "说白了就是计算两个向量之间的夹角，然后由余弦的图像就知道他们之间的相似度水平了。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c02e875c",
   "metadata": {},
   "source": [
    "# 四.嵌入矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d95fc0ec",
   "metadata": {},
   "source": [
    "<img src=\"./picture/26.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f25aee9f",
   "metadata": {},
   "source": [
    "和我前面的猜想一样的，目的就是为了训练这个嵌入矩阵。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea1a5d5f",
   "metadata": {},
   "source": [
    "# 五.学习词嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c235973",
   "metadata": {},
   "source": [
    "## 5.1比较复杂的算法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b4c4c1c",
   "metadata": {},
   "source": [
    "<img src=\"./picture/27.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe8997c4",
   "metadata": {},
   "source": [
    "就是将其特征向量取出来，然后放到神经网络中去训练，最后使用softmax函数来进行激活，最后可以算出\n",
    "最终的答案，而且通过你的训练集，神经网络会发现，必须将橘子苹果等这些水果放在一起才可以预测出最终的\n",
    "答案，因为他们的特征过于相似。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12db6a75",
   "metadata": {},
   "source": [
    "## 5.2简单的算法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec3e3afb",
   "metadata": {},
   "source": [
    "<img src=\"./picture/28.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55393e6c",
   "metadata": {},
   "source": [
    "其实完全可以不用将所有的词语完全嵌入神经网络中，如果只是为了预测，那么选择前面4个词语会比较好\n",
    "如果是为了训练，那么使用用大括号圈起来的3种方法会比较好，因为反向传播主要还是为了节省算力。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90a6914d",
   "metadata": {},
   "source": [
    "# 六.Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "333c62b4",
   "metadata": {},
   "source": [
    "<img src=\"./picture/29.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4334a77b",
   "metadata": {},
   "source": [
    "上述的是简化的词嵌入模型。首先找到上下词的特征向量，然后喂到softmax这个函数里面去，最后输出即可\n",
    "但是这个模型的缺点就是计算量过于庞大，所以要进行优化。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6474aba2",
   "metadata": {},
   "source": [
    "## 6.2 优化后的模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec1c9ff9",
   "metadata": {},
   "source": [
    "<img src=\"./picture/30.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45e6714a",
   "metadata": {},
   "source": [
    "为了应对这个问题，我们可以使用一个分级的分类器即可，比如出现概率较高的词会放到一起，这个样子可以\n",
    "大大的节省计算时间以及成本。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4674234",
   "metadata": {},
   "source": [
    "# 七.负采样"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59ce8bc6",
   "metadata": {},
   "source": [
    "<img src=\"./picture/31.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcefc4d0",
   "metadata": {},
   "source": [
    "上面的图像讲述了图像采样的具体操作：\n",
    "\n",
    "1.首先我们选取了文本中的orange 然后发现juice是他后面的一个词语，可以把他标记成正采样\n",
    "\n",
    "2，然后从词典里面随机抽取单词，记作负采样\n",
    "\n",
    "3.然后负采样的个数就是，如果数据集比较小，那就取5-20个，如果数据集很大，就取2-5个即可。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0c56dc8",
   "metadata": {},
   "source": [
    "## 7.2负采样的具体操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5af257c5",
   "metadata": {},
   "source": [
    "<img src=\"./picture/32.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a734de53",
   "metadata": {},
   "source": [
    "首先还是和前面的softmax一样，找到orange这个词的特征向量，这里的负采样其实就是将问题进行简化了\n",
    "全部变成了二分类的问题，说白了就是$k+1$个二分类的问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b415d62f",
   "metadata": {},
   "source": [
    "## 7.3 如何选取负样本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e391ea",
   "metadata": {},
   "source": [
    "<img src=\"./picture/33.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03960229",
   "metadata": {},
   "source": [
    "根据经验可以知道，一般就是这个词在文本中出现的词频的$3/4$次方除以所有词语词频的$3/4$次方的和就是这个词语可以杯选中的\n",
    "频率。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "461fb2e7",
   "metadata": {},
   "source": [
    "# 八.Glove词向量嵌入"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a69ad69",
   "metadata": {},
   "source": [
    "总是从难到容易，慢慢理解，这个方法会更加的简单。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13202cbf",
   "metadata": {},
   "source": [
    "<img src=\"./picture/34.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fed8e64",
   "metadata": {},
   "source": [
    "上面的公式十分的简单，说白了就是把第一个单词看成i，然后j表示的是这个单词出现的频次，到时候再看看吧\n",
    "有点看不太明白啊。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "804e7bfc",
   "metadata": {},
   "source": [
    "# 九.情绪分类"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d72266",
   "metadata": {},
   "source": [
    "## 9.1 平均的操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c50dbc3",
   "metadata": {},
   "source": [
    "<img src=\"./picture/35.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c30d49a1",
   "metadata": {},
   "source": [
    "这里要注意的一点就是，此过程是一个前向传播的过程，然后中间有一个平均的操作，无论你这个评论\n",
    "有多少个词语，都可以先把他进行一个平均操作，最后再使用softmax函数来进行输出，也是一个简化计算的过程\n",
    "同时这个平均的向量代表了这个评论信息的好坏。\n",
    "\n",
    "但是这个算法有一个特别大的缺点就是他没有考虑词序的问题"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f902380f",
   "metadata": {},
   "source": [
    "# 9.2 改进"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bc9067d",
   "metadata": {},
   "source": [
    "<img src=\"./picture/36.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b706105",
   "metadata": {},
   "source": [
    "说白了就是使用RNN来进行一个多对一的操作，这个样子就可以识别了，也就是说，上面那种一般不使用\n",
    "因为....懂的都懂"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b21866c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0a0ea888",
   "metadata": {},
   "source": [
    "# 十.词嵌入除偏。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c87efd5e",
   "metadata": {},
   "source": [
    "<img src=\"./picture/37.png\" alt=\"Drawing\" style=\"width: 500px;\" align=\"left\"/>  # 这个是保存图片的代码"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdb390a2",
   "metadata": {},
   "source": [
    "可以看到，由于社会的偏见，会使机器也得出一些偏见。所以要尽可能的消除这些bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e79ca32",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2930e62",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d51477",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
